Sometimes when a ChunkStateWitness arrives to be processed, the
block required to process it isn't available yet, we have to wait
for it. near#10535 implemented a
hacky way to do it by retrying the processing every 500ms until the
required block arrives. This PR will implement a proper solution, so
let's remove the hacky workaround.

The current function to verify state witness signatures
takes the epoch_id from the previous block. For orphaned
state witnesses the previous block isn't available, so
it's impossible to use this function. Let's add another function
that takes the epoch_id as an argument instead of fetching it
from a block. It will allow us to verify signatures of orphaned
chunk state witnesses.

Add a struct which keeps a cache of orphaned ChunkStateWitnesses.
It provides two methods that allow to easily add a ChunkStateWitness
to the pool, and then when a block arrrives, another method allows
to easily fetch all orphaned ChunkStateWitnesses that were waiting
for this block.

Let's split this function into two parts - one part which
tries to find the previous block for the given ChunkStateWitness
and another which processes the witness when the previous block
is available.
It will make handling orphaned witnesses easier - once the block
arrives, we can just call the second function to process it.

In the future it might also make it easier to process witnesses
before the previous block is saved to the database, which could
reduce latency.

Add a function which tries to determine in which epoch
the specified height is located.
It looks at the previous, current and next epoch around
the chain Tip and tries to figure out to which one of
them this height belongs.
It's not always possible to determine in which epoch
a given height will be, as the start height of the next
epoch isn't known in the current epoch, so the function
returns a set of possible epochs where the height might be.

For example if the tip is in the middle of the current epoch,
and the height is about epoch_length blocks higher than the tip,
this function would return [current_epoch_id, next_epoch_id],
as this height could be in either one of those epochs.

It will later be used to verify the signature of orphaned witnesses.
To verify the signature we need to know what is the chunk producer
who should produce a witness at some height, and for that we need
to know what's the epoch_id at this height.

When the previous block isn't available, we can't process the ChunkStateWitness
and it becomes an orphaned state witness, waiting for the required block to arrive.
In such situations the witness is partialy validated and then put in the `OrphanStateWitnessPool`.
It's impossible to fully validate the witness, as the previous block isn't available,
but we can do some basic checks to curb abusive behavior - check the shard_id, signature,
size, etc.

When a new block is accepted we can process all orphaned
state witnesses that were waiting for this block.
Witnesses which became ready are taken out of the pool and processed one by one.

There is no ideal size of the witness pool, node operators
might want to adjust it to achieve optimal performance.
Let's make this possible by adding a configuration option
which allows to control the pool size.

Add metrics which provide information about witnesses kept in the pool.

Metrics are increased and decreased using a RAII struct to ensure
that they're always correct, no matter what happens with the pool.

There was no way to get the index of a client with some AccountId,
let's add a method that enables it to `TestEnv`

Add a function that allows to wait until some chunk endorsement is sent out.

It will later be used to verify that a validator sends out endorsements
for an orphaned chunk state witness when its block arrives.

`stateless_validation` is a private module of `near_client`, so it's impossible
to use items defined there in integration tests. Let's expose some of them to
make their use in tests possible.

`run_chunk_validation_test` has a very useful function which allows
to determine a block producer at some offset. I would like to also
use it in my integration tests.
Let's add it to `TestEnv` so that other tests can also use it.

Add a function analogous to `get_block_producer_at_offset`, but for chunk producers.

near#10613 (comment)

In previous commit I added debug spans to avoid
writing witness_height, ... everywhere. I also
removed them from the ejection message, but that
was a mistake. The span contains information about
the witness that is being added, not the one that
is being ejected. Because of that the message about
ejecting a witness would contain invalid witness_height
and other parameters. Fix by using the correct parameters,
and names which don't collide with the ones in the span.

I imported near-o11y to use the function for initializing
a tracing-subscriber in epoch manager tests, but I didn't
import the corresponding nightly packages, so a CI check failed.
Let's fix this by importing the packages that the CI check
prosposed.

Ran
python3 scripts/fix_nightly_feature_flags.py fix
to satisfy the CI

While resolving a merge conflict caused by near#10646,
I accidentally removed a handler for chunk endorsement messages.
I didn't mean to do it, I wanted to only modify handlers for chunk state witnesses.
Let's undo the change and bring the chunk endorsement handler back.

This reverts commit 5cae3c3.

It was actually necessary to remove the endorsement message handler,
it doesn't compile otherwise 🤦